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The retrieval behavior and thermodynamic properties of symmetrically diluted Q-Ising neural 
networks are derived and studied in replica-symmetric mean-field theory generalizing earlier works 

(3JT), on either the fully connected or the symmetrical extremely diluted network. Capacity-gain pa- 

rameter phase diagrams are obtained for the Q — 3, Q — 4 and Q — oo state networks with 
•^C uniformly distributed patterns of low activity in order to search for the effects of a gradual dilution 

of the synapses. It is shown that enlarged regions of continuous changeover into a region of opti- 
mal performance are obtained for finite stochastic noise and small but finite connectivity. The de 
Almeida- Thouless lines of stability are obtained for arbitrary connectivity, and the resulting phase 
diagrams are used to draw conclusions on the behavior of symmetrically diluted networks with other 

£^ ' pattern distributions of either high or low activity. 

C' 
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I. INTRODUCTION 

i-rt ' The storage and retrieval properties, as well as the dynamics, of large attractor neural networks have been studied over some 

time and numerous results are now available. The presence of an exponentially large number of unwanted spin-glass-like states 
in fully connected networks may limit severely the storage capacity, the information content and the performance, in particular 
the retrieval quality of previously stored patterns which act as attractors in the dynamics of the network. Also, the size of the 
corresponding basins of attraction may be considerably reduced. Other tasks, as the categorization or generalization ability of 

^j ' a network, are also impaired due to the presence of such states and, except for a low storage ratio, that is for a reduced level 
of stochastic noise, a performing network is very likely to be trapped in one of those states preventing the occurence of finite 

_„ ■ overlaps with the patterns of interest. 

■^- ' The study of the equilibrium behavior of the Hopfield model with binary neurons and extremely low symmetric connectivity 

t-H , c suggested that |1[ , except for zero storage ratio a or in the absence of synaptic noise T, the properties of the model may be 
quite different from those of the extremely diluted network with asymmetric synapses, which has a trivial dynamics M. In 
particular, the equations for the order parameters are equivalent to those of the Sherrington and Kirkpatrick (SK) spin-glass 

^ i (SG) model [|. 

It has been shown, nevertheless, that strong symmetric dilution of synapses in the Hopfield model MM may considerably 
reduce the stability of spin-glass states. Indeed, in the limit c — > 0, the retrieval states are globally stable in their whole domain 
of existence of the (a, T) phase diagram, thus leaving only locally stable SG states [jf, in contrast to the situation in the fully 
connected Hopfield model M. It has also been shown that already a gradual dilution reduces the stability of the SG states in 
the region where they compete with the retrieval states and that the storage capacity of the network is increased M. These are 
important results on the performance of a network which also suggest that the basins of attraction of the memory (retrieval) 
q states should be increased in symmetrically diluted networks. 

Networks of Q-Ising neurons in various architectures have been studied over some time, in particular, fully connected, layered 
feed-forward and extremely dilute networks with asymmetric connectivity [8 — 15]. They are of interest for possible hardware 
applications and an eventual biological modeling, because of the increased flexibility of the states of the network to account 
for complex neural behavior. It has been suggested that an attractor neural network model with multi-state neurons may 
describe the short term memorization performance of the CA3 region of the hyppocampus in both the brain of primates and 
in the human brain, and results of numerical simulations for the selective performance of a network with asymmetric synapses 
and small connectivity (c = 0.2) are now available pa]. However, complete phase diagrams that give a global picture of the 
performance of a network and a physical explanation of why biological networks seem to prefer a low connectivity are still 
missing. Attractor network models with a Hebbian type of learning rule may also serve to account for long-term memory in 
the brain (Tt) . 

Both the parallel dynamics and the equilibrium properties of extremely diluted networks with symmetric connectivity between 
multi-state Q-Ising neurons have been studied in recent works |18|JlS|| . The symmetry of the synaptic connections allows for a 
detailed balance assumption in the dynamics and enables one to perform full analytic calculations of the equilibrium properties 
of a network. This is particularly convenient in the search for phase diagrams. So far, there are no results, to our knowledge, 



e 

-6 



- 1—1 

X 



concerning the memorization performance of symmetrically diluted Q-Ising networks with small but finite connectivity and 
low-activity patterns, which characterize a biological network. 

The purpose of the present work is to consider the retrieval performance of an attractor neural network modeled on some of 
the features of a biological network. We study here the equilibrium behavior of a symmetrically diluted Hopfield model with 
finite connectivity and low-activity units in Q-Ising states. Multi-state networks are known to have complex properties and 
our main aim here is to find out how that behavior is changed by a finite dilution of the synapses and the low activity of the 
embedded patterns. We consider explicitly a Q — 3, Q = 4 and a continuous Q = oo-state network. 

The outline of the paper is the following. In Sec. II we present the model and show how the Hamiltonian becomes the sum 
of an effective Hopfield-model Hamiltonian and that of a Sherrington-Kirkpatrick-like spin-glass model. The replica-symmetric 
mean-field theory for the model and the relevant order parameters are derived in Sec. Ill and explicit expressions are presented 
in the Appendix. The results for the phase diagrams and the retrieval performance are discussed in Sec. IV, and our conclusions 
with a summary of the results can be found in Sec. V. 

II. THE MODEL 

Attractor neural networks are dynamical systems. Consider a network of N neurons, i — 1, . . . ,N. At the time step t, the 
state of neuron i is described by the variable Si(t), that can be in any one of the Q Ising states 

1 , 2(fc-l) 

in the interval [-1,1], for k — 1, ...,Q. A macroscopic set of p independent and identically distributed random patterns 
{£?; fx — 1, . . . ,p;i — 1, . . . , N}, with p = acN, is embedded in the network by means of a Hebbian-like learning rule, specified 
below. Here, a is the storage ratio per connected site and c is the connectivity of the network. Every bit of each pattern can 
be in any one of Q equally spaced states, also in the interval [-1,1], which are assumed to have zero mean and variance 

((tf) 2 )=a<l. (2) 

This is a measure of the size of the patterns, describing their mean activity, that accounts for bits that either are turned off or 
depressed. 

The first task to be performed by the network is to attain a finite storage ratio a and this ratio can be optimized for patterns 
of a given size by means of an appropriate tuning of the states of the network, discussed below. The second aim is to reach a 
sufficiently small Hamming distance 

dH(^,<r)=N- 1 Y f (&-Si) a (3) 

i 

between the network state {Si} = (Si, . . . Sn) and a given pattern {£?}, as a measure of the retrieval performance of the 
network. This depends on both the overlap 

m" = (aiV)- 1 ^Cr^ (4) 

i 

and the dynamical activity of the network, an = (aiV) -1 y\ Sf. 

We consider next the dynamics of the partially connected model. For a given configuration {Si} of the network, the local 
field acting on neuron i is given by 

hi{{S i }) = J2 J tj s i> ( 5 ) 

i 

where Jy is the synaptic coupling between neurons i and j. We assume that the synapses of the network are symmetrically 
diluted being left with a finite connectivity c (the fraction of connected neurons) which, eventually, may become vanishingly 
small. Specifically, the synapses are of the Hebbian-like form, 

for our non-sparse network, in which ((£f )} = 0. Here, {dj = Cjt; i,j = 1, . . . , N} is a set of independent identically distributed 
random variables, such that Cij — 1 with probability c and zero with probability 1 — c, while cu — 0. Thus, the symmetric 
dilution introduces an additional randomness both into the dynamics of the network, which becomes non-trivial even in the 
extremely diluted case due to feedback loops Jl8( , as well as in the thermodynamics. We are interested, in the following, in the 
case of a large connected network, in which cN is very large. 



The state of each neuron is updated asynchronously according to a Glauber (single spin-flip) dynamics H in which the 
transition probabilities are given by 



p(5 i (t+At)=a fc |{5 i (t)}) = ^ yF r ,':.::!::!„ . (7) 



^exp^HMI&W})) 
where (3 = 1/T is the inverse synaptic noise (temperature) and the single site energy, tj(s\h), is given by 

Cj(s\h) = -hs + 8s 2 . (8) 

Here, 9 is a tuning parameter favoring local states of small dynamical activity. In the absence of stochastic noise, the deter- 
ministic evolution of the system follows the updating rule 

Sj{t + At) = e dv n(hj(t)), (9) 

where Qdyn(x) is the non-decreasing step function, for finite Q, 

Q 

Qdyn(x) = ^ Ok[Q{0{o k + 1 + O k ) - x) - 0(6((T k + CT h -l) - X)] (10) 

with (To = — oo and <jq+\ = +00, in which Q(x) = 1, if a; > 0, and zero otherwise. Clearly, 6 is a threshold parameter since 
the state of neuron j assumes the value a k given by Eq. (fil) if the local field hj is bound by a k + (Tk-i < hj/6 < cti, + crk+i- 
The width of the intermediate states with constant a k for 1 < k < Q (that is, excluding the limiting values of a k = ±1), is 
given by 48/(Q — 1). Thus, the width of the zero state for the three-state network studied below is 29. In the limit Q — » 00, 
the input-output function becomes the piecewise linear function 

Odyn(x) = sign(z)min ( LJ, 1 J , (11) 

where min(a;, y) means the minimum between x and y. The slope of the linear part, 1/26, is the gain parameter of the continuous 
network. 

Both, the locally stable states of the dynamics, as well as the globally stable states that characterize the equilibrium 
thermodynamic properties of the diluted network, that follow from the above dynamics, are described by the Hamiltonian 

h = - J2 4 s * s i + e J2 s > - (12) 

where the first sum is over all distinct pairs of neurons (i, j). Eventually, a field-dependent term, hi ^ . ^jSi, may be added to 
generate the overlap with a specific pattern, and this will be implicitly assumed below. 

We adapt next the procedure of Viana and Bray EQ] for diluted spin glasses in order to deal with the random dilution. In 
distinction to their case, which is that of a strongly interacting spin glass, we have here a diluted network with randomness in 
the patterns and weak interactions between units. The latter allows for an exact truncation to the relevant order in 1/cJV that 
is sufficient for the mean-field calculation of the following section. 

Consider the disorder dependent part of the Hamiltonian in the exponentiated form 

G ({Si}) = expiP^jfjSiSj] . (13) 

Given a fixed set {££* } of patterns embedded in the network, we first have to build up a finite connectivity between units. 
That is to say, we have to find a network such that the mean of cy is precisely c and, to this end, we have to perform first an 
average over these random variables. The average over the random patterns, that is necessary to evaluate the performance of 
the network, comes at a later stage. The configurational average over the set {cij} of the n times replicated function Gfj({Si\) 
becomes 

(G n ({Si})) c = exp{^ ln(l + c(exp09Jy £ S?Sf) - 1))} , (14) 

where Jij denotes the value of jA for Cij — 1. We are interested in dense networks, that is, c = O(l) and large cN, letting 
eventually c — *■ after the thermodynamic limit. For finite a = p/cN, the weak Jy = 0{^Ja/cN) may be used to expand the 
logarithm to next-to-leading order in Jij. Using the law of large numbers to deal with {Jij) 2 we obtain 

<G^{^})) c =/exp{/3^J^Sr^}\ , (15) 



in which the effective coupling J, e - is given by 



/< 



(1=1 

The first term is the Hebbian synapsis of the fully connected network with multi-state patterns of activity a and 8ij is a Gaussian 
random variable with zero mean and a pattern-independent variance A 2 /N = q(1 — c)/N\ the brackets on the right-hand side 
of Eq.(15) denote an average over this random variable. This generalizes an earlier result by Sompolinsky |2lJ showing that 
dilution appears as a synaptic noise. 

Thus, the symmetric dilution introduces an effective Hamiltonian, in the large cN limit, 

H« ff = - Y, J t"SiSi + 0J2 Sf , (17) 

where the first term is a sum of a Hopfield-model Hamiltonian and a kind of SK-model term. This is used in the following 
section to derive the mean-field theory for the model. 

III. MEAN-FIELD THEORY 

We consider now the mean-field theory for finite a and for any connectivity c— O(l). The averaged free energy per connected 
site is given by 

/(£)=- lim J-/ (In Ze ff (13)) {} ) , (18) 

with the averages first over the Gaussian noise in the synaptic interactions and then over the pattern distribution. The effective 
partition function is then given by 

^e//(/3) = ^exp(-/3 J ff e// ), (19) 

and this is used in the replica method to perform the random averages by means of 

(M//(flw) {4(I} = **± (((ssr/ww^-i) • (») 

The essential point is then the calculation of the averaged replicated partition function. Assuming, as usual, that a finite 
number of patterns £f is condensed, with finite overlaps with the state of the network, we perform the random noise average 
and sum over the uncondensed patterns to obtain 

-A 2 /3 2 



{S?} a,0 i 

x | n ^< exp{ _^ E(<)2)+npm 

a a. 

x /e X p{/?^ [mUiS? - e(ST) 2 ]V) ■ (21) 

\ i,ot I {^f} 

where, the sum over {S? } counts the configurations for all possible replicas a. The dilution appears only in the first factor and 

exp(/3n P F) = [] ( exp{ A £ ^ S? 8$%$ } \ (22) 
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is the average over the uncondensed patterns. 

For both, the calculation of the function F and the linearization of the quadratic form in the dilution term, we introduce 
the replica matrix elements 

q a p=Q a p=jj^S?Sf iia^p, (23) 



of the SG order parameter and 

q a = Q aa , (24) 

which is the dynamical activity cld of the network. The latter is one only in the case of binary units and, in general, q a p < q~ a < 1. 
In the thermodynamic limit N — > oo, we obtain [Ea 

n/3G = -itrln(l-/?q)-i/3trq. (25) 

where q is the matrix of elements q a p and q a . Introducing as usual the overlap parameter r a/ g associated to the correlation 
between the overlaps of the patterns that do not condense, we restrict ourselves to replica symmetry, in which 

m v = m u a (26) 

q = q<±fi (27) 

a D - q~ a (28) 

r = r aP . (29) 

The free-energy per connected site, in the thermodynamic N — ■* oo limit, then follows as 
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/ Vz ln^exp(/3H, 



t-M -c)(^ + ^)-i( /©zln^exp(/W e// )) CiO) 

' w {S} / un 



where Vz = dzexp(— z 2 /2)/\/2n is a Gaussian measure, and x = P{ a D — q), given by (3 ^.({S 2 ) — (Si) 2 ) /N , is the susceptibility 
of the network. The new, site independent effective Hamiltonian H e ff, is given by 



H e ff = S \Y1 m "^ + V^* -QS\ , 



( 31 ) 

V / 

in terms of which the thermal averages are denned as 

(S n (z)) = ^ \ "' . (32) 

Note, incidentally, that the explicit term in a(l — c) in the free energy comes from the variance of the Gaussian random noise 
due to the dilution. 

The order parameters that describe the performance of the network are given by the saddle-point equations 

m « = \U" Jvz{S(z))\ , (33) 



Vz{S{z)Y) , (34) 

and the susceptibility becomes 

r Vzz{S(z))\ , (35) 

in which (S(z)), for a given Gaussian noise z is given by Eq.(32). Noting that the effective Hamiltonian TC e ff is formally the 
same as either that of the fully connected or the extremely diluted network, with a different stochastic Gaussian noise and 
different effective threshold 0, for each connectivity c, our equations for m v , q and \ wn l De formally similar to the equations 
for both these networks, and explicit expressions are given in the Appendix. 
The parameter r follows from the algebraic saddle-point equation 

r=q{ w 1 w +1 -^ ) ' (36) 

and 



= 9-?f(l+^-), (37) 

A 

is the effective width of the intermediate states. Eventually, depending on the state of the network specified by the dynamical 
activity an and the SG order parameter q, 9 may become negative, favoring an order with large absolute values for S. Whenever 
this is the case the network acts, at zero temperature, as a binary network with zero threshold, as will be seen below. The 
combination ^ m„£" + ^/arcz, in which the second term is the Gaussian noise due to the macroscopic number of uncondensed 
patterns, can be viewed as an effective random field that will influence the network performance through a competition with 
the effective threshold 9. 

Having performed the thermodynamic limit, one may now allow any value for the connectivity within < c < 1, including 
the limit c — » 0. It can easily be seen that one recovers for /(/9) both the mean-field free energy for the fully connected network 
when c = 1 Jl5| and that for the extremely dilute case, when c — > J19J. In this limit, re = q and 9 — 9 — «x/2, whereas 
for c = 1 we have, r — q/(l — X) 2 and 9 — 9 — ax/2(l — x)i a l so m agreement with the known results. These relations are 
valid for any temperature parameter 1//3. Thus, we have the complete form for the replica-symmetric free energy for arbitrary 
connectivity. 

Note, incidentally, that re — q not only in the above limit but also for a = 0, in the f3 — * oo limit, for any connectivity c, 
(that is, for any architecture) and for all Q, and this is based on \ ~~ * m that limit. Otherwise, the susceptibility remains 
finite, even at zero temperature, since q — > an, when /? — ► oo, while at finite temperature we have, in general, q < ao- 

The limit of stability of the replica-symmetric solution comes from the study of quadratic fluctuations of the free-energy in 
the vicinity of the symmetric saddle-point. Following the de Almeida and Thouless (AT) analysis J23J, we obtain 

^([v z [(S\ Z ))-(S( Z )f] 2 ) <1, (38) 

1 \J I {$<} 

as the stability condition for the replica-symmetric solution. This equation is to be solved together with the saddle-point 
equations for the order parameters. 

The formal results obtained so far are valid for any finite number of condensed patterns with finite overlaps with the state 
of the network. We are mainly interested in this work in the retrieval performance with a single condensed pattern, and this 
will be discussed in the following section. 

IV. RETRIEVAL AND THERMODYNAMIC PROPERTIES 

For the retrieval of a single condensed pattern, say f 1 , we have m„ = m<5„i, and omit the index 1 from now on. We consider 
separately the results for Q = 3, Q = 4 and Q = oo that follow by solving the saddle-point equations and restrict ourselves to 
the simplest case of uniformly distributed patterns. 

A. The three-state network 

In the Q — 3 case, the patterns take the values ±1 with probability a/2 and with probability (1 — a), in which a = 2/3 in 
the case of uniformly distributed patterns. The effective transfer function Sp(h,9) = {S{z}) that follows from the average in 
Eq.(32) is given by 

Se(h,6) = - S^ (39) 

\ exp(/30) + cosh(/3ft) 



which becomes 

Soo(/i s ,e)= lim Sp{h s ,6) = sgn(h s )e(\h s \-6), (40) 

in the zero temperature limit. Clearly, when 9 < 0, the network acts as a binary network at T — 0. 

In the cases of the fully connected Jl5| or the extreme symmetrically diluted network |l9| , explicit closed form expressions 
that signal the appearance of either a retrieval or a spin-glass phase, at zero temperature, have been obtained in the 9 — > 
limit when 8 < 0. These are particularly useful to understand the low-threshold behavior of the phase diagrams. In the present 
case, of a network with finite and less than complete connectivity, closed form expressions for the onset of the ordered phases 
cannot be obtained and one has to resort fully to numerical solutions. 

Nevertheless, the zero-temperature behavior of the thermodynamic transition can easily be analyzed as in previous works, to 
demonstrate that the retrieval state corresponds to the most stable phase, despite the presence of a spin-glass and a paramagnetic 
phase (which is a frozen zero-spin state). Indeed, the physical free energy, / = — /(/3) at zero temperature becomes 



f---m - -xrc + Brc. (41) 

Since the susceptibility vanishes in the a — > limit when T — for any of the three phases, and at the same time re — > q, the 
retrieval free energy is the minimum whenever 9 < 1/2. Note that the susceptibility of the paramagnetic phase, 

X = W , (42) 

exp(/36>) +2 y J 

also vanishes in the zero temperature limit for finite a and 9 converging to 9, ensuring a minimum retrieval free energy for 
small a. 

Although much emphasis is often made on the thermodynamic transition to globally stable retrieval states, which have the 
lowest free energy, it is worth keeping in mind that neural networks are dynamical systems with accessible locally stable retrieval 
states, in particular in the presence of some amount of noise. As far as the performance of the network is concerned, these are 
the most interesting states and they usually appear for higher values of a M . 

We are interested here in the characteristic features of the phase diagrams and the specific performance of the network. To 
see the effects of a gradual change in the connectivity, we show in Fig. 1 the (a, 9) phase diagrams for T — 0. The full lines 
represent the phase boundaries where the locally stable retrieval states appear at the critical storage ratio a c , whereas the 
long-dashed lines indicate the thermodynamic transitions to the globally stable retrieval phase. The SG phase appears to the 
left of the short-dashed lines representing the boundary to the paramagnetic state. To distinguish in what follows the transitions 
involving locally stable states from the thermodynamic transitions, we refer to the former simply as retrieval transitions. As 
long as the connectivity remains finite, all the transitions are discontinuous and, as usual in connected networks, the SG state 
is globally stable only above the thermodynamic transition nj. 

Consider first the case of the fully connected network, with C — l, which has been redone and completed here for the purpose 
of comparison. There are two retrieval regions, I and II for small a, separated by a sharp phase boundary, and this is the 
case both below and above the thermodynamic transition. The first is a region of non-optimal performance characterized by 
a moderately large Hamming distance which decreases with increasing 9, whereas in region II the Hamming distance is small, 
dropping discontinuously at the phase boundary between the two regions with optimal network performance along the dotted 
line. 

The situation should change with decreasing connectivity, even at zero temperature. Due to the synaptic noise produced 
by the dilution, given by the variance A 2 /7V = a(l — c)/N, one now expects an end to the discontinuous transition between 
the retrieval regions I and II at a c-dependent critical point for finite a, below a c . This starts to appear for c ~ 0.63, with 
9 ~ 0.325 and a ~ 0.0282, as shown in the inset of the figure for the retrieval transition. A critical point also appears at the 
thermodynamic transition and in Fig. 1 we show the results for c = 0.5 and c = 0.25. There can be now a continuous transition 
with increasing 9 induced by stochastic noise between the non-optimal and optimal performance retrieval states above the 
critical point suggesting the presence of increasingly larger regions of continuous changes. The interesting point is that this is 
a feature that already starts to appear for finite and intermediate connectivity, between that of the fully connected network 
and the c — > limit. 

To understand the role of the threshold one may use as a guide the case of vanishing stochastic noise, with a — ► 0. When 
9 is small, the state of a unit will be essentially Si — ±1, with an overlap m^ — (aA r )~ 1 ^ |£f | = 1 in region I for uniformly 
distributed three-state patterns £<*, that take the values —1, and +1. Since q = 1, the Hamming distance in this region will 
be <1h{I) — 1/3. On the other hand, as 9 becomes larger, the state Si — becomes increasingly important and, despite the fact 
that Si = £t also yields an overlap m^ — 1, the SG order parameter q is now reduced to the activity 2/3. Thus, the resulting 
Hamming distance <1h{II) will be vanishingly small. Note that these results do not depend on the connectivity and they are, 
therefore, independent of the architecture of the network. This also follows from the zero-temperature saddle-point equations 
as can easily be checked with the Appendix. In the case of a finite non-zero stochastic noise, instead, the performance of the 
network becomes explicitly dependent on the connectivity, but the overall qualitative dependence on 9 below the critical phase 
boundaries is expected to follow that at a = 0. 

The critical storage capacity now increases with decreasing connectivity and the presence of two comparable maxima for a c 
is only a feature of intermediate c. Indeed, as the synapses are further diluted a single maximum is left, albeit with a shift 
to higher values of 9. Finally, it is also worth noting that, in the c — > limit, the retrieval state is the globally stable phase 
everywhere below the critical a c line and to the left of the globally stable paramagnetic phase, despite the relatively large 
stochastic noise due to the presence of spin-glass states in most of this region. 

The zero-temperature results presented so far are not stable to replica-symmetry breaking perturbations but it is expected 
that most of the features described here will be present at already a small but finite temperature above the AT line, shown as 
dash-dotted in Fig. 2, for 9 = 0.2. The full lines again represent the transitions to the locally stable retrieval states and the 
thermodynamic transitions are not shown. Note that, even for small connectivity, there is only a low synaptic-noise region in 
which the network is not stable to replica-symmetry breaking perturbations. For a larger 9 = 0.5 we expect a similar behavior 
for the transition to the locally stable retrieval state with decreasing connectivity as that found before for the fully connected 
network, but still quite different from the behavior for lower 9 ]15|. 



B. The four-state network 

In the case of Q = 4 the phase diagrams are more involved. The patterns are assumed to take the values ±1 with probability 
5/2 and ±1/3 with probability (1 — 5)/2, in which a — (9a — l)/8, where a — 5/9 for uniformly distributed patterns. We 
consider in the following the zero-temperature behavior of the network and begin with the fully connected case as a guide. We 
recover precisely the retrieval phase boundaries found before ]15| , 

Similar results with exclusively sharp phase boundaries, between now enlarged ordered regions, are found for somewhat 
lower connectivity, as shown in Fig. 3(a) for c = 0.5. For low a, we find three different ordered retrieval ferromagnetic phases 
FMza, FMi and FMa, in a previous notation [|15| and characterized below, separated by discontinuous phase boundaries and 
in which A = 2/5 for uniformly distributed patterns. As a increases, these phases disappear discontinuously at the critical 
phase boundaries o c into the SG phase. The three phases correspond to possible locally stable states which become globally 
stable at the thermodynamic transition for lower a, not shown in the figure for simplicity. Which of the locally stable states is 
actually reached in the dynamic evolution of the network will depend, as usual, on their basins of attraction and the choice of 
the initial state. 

One may consider four regimes in the a — > limit. First, when 9 is small, that is within the phase FMza, mainly the high 
activity states are favored with Si = sgn(£i) with an overemphasized overlap m — 6/5; the spin-glass order parameter and the 
Hamming distance become q = 1 and du = 2/9, respectively. For larger and intermediate 9, there should be a phase, called 
FM\, characterized by states of the network that follow essentially the patterns, with Si — £» and an overlap given by m = 1, 
the spin-glass order parameter q — a — 5/9 and a vanishing Hamming distance. Between the small and the intermediate 9 
regimes there could be a coexistence region of the phase FMza with the phase FM\ where the states of the network start to 
recognize the full structure of the patterns. Finally, there should be a phase characteristic of the large 9 regime in which mainly 
the intermediate states with Si = |sgn(£i) are activated leading to a performance with m = 2/5, q — 1/9 and again du — 2/9. 
This is the phase FMa, which should have an overlap with the phase FM\ at intermediate 9. These expectations have been 
confirmed by means of the solutions to the saddle-point equations in the a — » limit and some of the results can be found in 
earlier work (T||. The four regimes are given by 9 < 1/4, 1/4 < 9 < 3A/4, 3A/4 < 9 < 3/4 and 3/4 < 9. They do not depend 
on the connectivity and are therefore independent of the architecture of the network, in accordance with earlier results either 
on the fully connected or the extreme symmetrically dilute network ]15| , |l9[ . 

On the other hand, we confirm the symmetry of the limiting a c for 9 — > oo and for 9 = 0, in accordance with earlier results 
[ JL5| . We also find that the optimal performance line appears within the FMi phase and that the network has a relatively high 
performance with a small Hamming distance in that phase, with an overlap at the critical phase boundary that is 0.8 of that 
at a = 0. 

When the connectivity is reduced to c ~ 0.48, the distinction between the phases FMza and FM\ starts to disappear, as 
shown by the enlarged gap in Fig. 3(b), allowing for a continuous change into the high performance phase for intermediate 
a. Note that there is still a discontinuous phase boundary between the phases FMza and FMa and that the presence of this 
phase boundary is important in order to inhibit the transition to the low performance phase FMa- 

Furthermore, we still find four regimes for low a and that the three main retrieval phases, FMza, FMi and FMa, end 
discontinuously at the critical phase boundary a c . The optimal performance line is still purely within the FM\ phase, as in 
the previous case, and the network has a high performance up to a c , with an overlap close to one on the phase boundary, for 
9 around 0.5. 

As the connectivity is further decreased to c = 0.1 we find the phase diagram shown in Fig. 3(c) with the three main retrieval 
phases that disappear discontinuously at a c and the four low-a regimes discussed above. The distinction between the phases 
FMza and FM\ disappears now at lower a and the optimal network performance in the central phase FM\ can be reached 
continuously within a considerable range of a from the FMza phase. As in the previous cases, there is a coexistence region 
between the phases FMza and FMi, now only for small a. Moreover, there is no need now for a specific choice of threshold 
parameter 9 in order to access most of the high performance domain of the network. Incidentally, note that the continuous 
retrieval phase boundary for the common phase FMza and FM\ is similar to that found for the fully connected network with 
pattern activity a = 7/9 ]l5| . We have no further insight, at present, of this feature. 

Finally, in order to check the overall simplification of the phase diagrams that appears with decreasing connectivity, we also 
present results for c = 0.01 that are shown in Fig. 3(d). The phase boundaries are still lines of discontinuous transitions and 
the distinction between the four regimes is restricted to even lower values of a. There is now a considerably larger region of 
continuous changeover from the phase FMza to the phase FM\, with access to optimal performance, without the need of a 
fine adjustment in 9. 

C. The continuous response network 

In the case of Q = 00 we again consider uniformly distributed patterns between —1 and 1, implying that a — 1/3, and restrict 
the results to the zero temperature case. The discontinuous transitions to the ordered phase are shown in full lines in the a — 9 
phase diagram for decreasing connectivity in Fig. 4, where we omit again the thermodynamic transitions and the long-dashed 
lines indicate now the onset of the binary-network behavior. Note that the disappearance of the ordered phase takes place at 



8 = 1/2 for any finite connectivity, as in the case of the fully connected network and in networks of different architecture, like 
the extremely asymmetric diluted and the Q-Ising layered network [ |15[|l3| , |l4| - in the case of the extremely diluted networks, 
the retrieval phase boundaries have a reentrance for 8 > \ M. This seems to be a feature of the c — > limit, as one can see 
from our further results for the connectivity dependence of the maximum storage capacity a ma x and the corresponding 8 ma x, 
both shown in Fig. 5. 

As in the case of both the fully connected and the symmetrical extremely diluted network, and in contrast with the Q — 3 
state network, we find that even at zero temperature most of the retrieval regions for different c are stable to replica-symmetry- 
breaking perturbations, that is for 8 above the AT lines. This includes the maximum storage capacity and it follows from a 
positive replicon eigenvalue for this case obtained from Eq.(38), 

X R = l-™([ Vz\ = i_^£X, (43 ) 



where \ m the susceptibility for the continuous network presented in the Appendix. The AT line is given by Xr — and, 
again, in both the c = 1 and the c — *• limit, in which re = q, this result coincides with that for the fully connected and the 
symmetrical extremely diluted network |15[|l9[ . 

V. SUMMARY AND CONCLUDING REMARKS 

We derived in this work the replica symmetric mean-field theory for Q-Ising attractor networks with low-activity patterns 
and arbitrary symmetric dilution of the synaptic connections. We extended earlier studies on the retrieval behavior and 
thermodynamic properties of either fully connected or symmetrical extremely diluted Q-Ising neural networks with low activity 
patterns, in order to study the effects of a gradual dilution of the synaptic connections guided by the motivation that neurons 
in biological networks of associative memory are neither fully connected nor very sparsely linked to other neurons. We are 
mainly interested in the nature of the phase transitions to locally stable retrieval states and in the role that synaptic dilution 
has in either reducing or destroying sharp transitions motivated by the plasticity of biological networks. In the context of the 
networks studied here, we focus mainly on the dependence of the retrieval properties on the threshold or gain parameter 6 
for decreasing connectivity c and ask to which extent can a network go from a given locally stable retrieval state to a nearby 
high-performance state without crossing phase boundaries of discontinuous transitions. 

In order to answer that question, one has to look for appropriate phase diagrams that were obtained here in replica-symmetric 
mean-field theory. Since deviations from that theory are very small and appear only in a small region near T = 0, we may 
still draw relevant conclusions from those diagrams. The explicit phase diagrams obtained in this work apply to uniformly 
distributed patterns and to networks with arbitrary symmetric dilution. The behavior of both the fully connected and the 
extremely diluted network are recovered when c — 1 and c — * 0, respectively, in that our general saddle-point equations become 
identical to those for either case that have been obtained before llaJlSh. 

We find that common features of the a — » limiting behavior in either fully connected or symmetrical extremely diluted 
networks also appear for arbitrary finite connectivity c. This confirms the expectations of earlier works that pointed out the 
architecture independent nature of some properties |15[|l9fl . Among these is the particular 8 — 0.5 where the thermodynamic 
transition ends in the Q — 3 state network and where the optimal performance takes place for low a, both for T = 0. The 
common limiting T in the (a, T) phase diagram for varying c is a further property of this kind, as well as the four distinct 
domains in the Q = 4 state network and the 8 — 0.5 limiting threshold for the Q — oo network at a — 0. 

The main dependence of the behavior of the network on the connectivity arises for finite a. For both odd and even Q, we 
find that a common feature that appears with an increase in the dilution of the synaptic connections is to suppress selected 
sharp phase boundaries of discontinuous transitions that make the optimal performance domain readily accessible to a wide 
region of low-threshold locally stable retrieval states. Note that, on the other hand, the sharp boundary for the Q = 4 state 
network between the low performance phases FM3& and FM& survives synaptic dilution, at least to quite an extent. These 
features of the network for small but finite connectivity appear long before the extremely diluted limit and they should be of 
considerable interest. 

Concentrating, for simplicity, on the Q — 3 state network we also found that the boundaries between thermodynamic 
transitions are suppressed by an increase of the synaptic dilution, and expect a similar behavior for the Q = 4 state network. 

The results of our work may be used to infer the behavior of other networks. Since the fully connected network is strongly 
sensitive to pattern activity, one may consider other than uniformly distributed patterns MM. There are, essentially, two kinds 
of phase diagrams in that case about which we can make definite predictions. One is the type of phase diagram for patterns of 
relatively large activity that has mostly a decreasing phase boundary with increasing 8 and an optimal performance line that 
appears only at small 8. The other type, that appears for medium or small pattern activity, has an optimal performance only 
at intermediate 8, like the cases shown in Fig. 3, and both types appear for Q — 3 and Q = 4, while only the first type seems 
to appear for Q = 00. We expect that the main effect of a finite synaptic dilution on the first type of phase diagram is simply 
to shift the retrieval phase boundaries upwards towards a larger a c . In the second type of phase diagrams, however, we expect 



also a disappearance of the discontinuous phase boundary between the FM3& and FMi phases, in essentially the same way we 
found in the present work, allowing for a smooth changeover from states of non-optimal to those of optimal performance. 

The main result of this work, that partially connected multi-state Hebbian networks can attain near-optimal performance 
without a fine tuning of neuron activity may be a simplified statistical mechanics explanation of why biological memory networks 
seem to prefer low-activity patterns between partially connected neurons. Of course, biological networks have asymmetric 
synaptic connections which may lead through a dynamic evolution to different stationary states, the search of which is certainly 
an interesting issue that deserves a separate investigation, currently in progress. 

The study of the effects of symmetric synaptic dilution may be extended to other problems that deal with associative memory, 
like the categorization problem as a classification task in Q-Ising networks 124] . This has been done recently for Q = 2 P5J and 
there is work in progress for general Q [M. 
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VI. APPENDIX 



We present here, for completeness, the explicit expressions for the saddle-point equations obtained for the symmetrically 
diluted network with arbitrary connectivity c and uniformly distributed patterns, for Q — 3, Q = 4 and Q — oo. 
For Q — 3 taking patterns ±1 with probability a/2 and with probability (1 — a) we have 



m— I T>z Sp(m + y/arcz, 9) . 



(44) 



q— I Vz \aS{3(m + y/arcz, 9) + (1 — a)Sp{y/a.rcz, 9)\ 



(45) 



X = — 1= I Dz z \aSj3{m + y/arcz, §) + (! — a)Sp(y/arcz, §)] , 



(46) 



with Sp(h,6) given by Eq.(39). These equations reduce to the equations for the extreme symmetrically diluted network, as 
well as for the fully connected network, in the c — > limit and c—1, respectively. The same applies for the cases of Q = 4 and 
Q — oo, presented below. 

In the zero-temperature limit (/3 — > oo), the integrations over the Gaussian variable z can be done explicitly. In the Q — 3 
case we obtain, for 9 > 0, 



,, m + 9 m-9 

erf (^5=)+ erf (^=) 
V2arc v^a^c 



(47) 



9 = 1 



erf( 



m + i 



\/2c 



■erf( 



s/2c 



=) + (1 - o)erf(-^=) 



(48) 



X = 



27rarc 



, (rn + 0) 2 N , (m-6>) 2 , , , 9 2 , 



(49) 



with the relation between r, g and x given by Eq.(36). 

For the Q = 4 state model with states and uniformly distributed patterns that take the values — 1, —1/3, +1/3 and +1, 
a = 5/9 and we obtain 



m=-IVz 



Sp(m + y/arcz, 9) + S/3(— + \/arcz, 



(50) 



1=- 2 IVz 



S 2 g (m+V^z,9) + S 2 (j 



/ I 

(-- + Votrcz, 



(51) 



1 



IJarc 



Vzz 



Sp(m + ^/arcz, 9) + Sp(— + y/arcz, 9) 



where 



S p (h,9) = 



smh((3h) + | cxp(8/36>/9) sinh(/3/i/3) 



cosh(/3h) + exp(8/39/9) cosh(/3h/3) 
In the zero-temperature limit 

S^h, 0) = sign(h) [l + 29(\h\ - 49/3)] , 
and the above equations yield, for positive 9, 
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1 

To 



r ,3m + 40. .,3m — 49. m 

erf ( o /o ) + erf( o as ) + erf (^7=) 

m + 49 m~49 m , 

erf ( — -==) + erf ( — ) + erf 



3v2arc 3\/2arc 



3V2arc 



(52) 



(53) 



(54) 



(55) 
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q=l- 



-,3m + 40. .,3m -40, r ,m + 46, ., to - 40 , 

erf( — ) — orf ( — ) + erf ( — ) — erf ( — ) 

3y2arc 3\/2arc 3\/2arc 3v2arc 



(56) 



3^ 
+ exp(- 



exp(- 



(m + 46>) 2 
18arc 



(3m + 46>) 2 



18arc 



+ exp(- 



(3m- 40) 2 , . m 2 , 

' 18arc )+eXp( "2^ ) 



+ exp(- 



(m - 46») 2 



is,,., '' +exp ( isarc^ 



(57) 



Finally, in the zero-temperature limit for Q = oo and uniformly distributed patterns between —1 and 1, implying a = 1/3, 
we obtain for 6 > 0, 

-l 

■m J- I / stiff _. 

(58) 



— 5 y_,«« 



,„ m£, „/„/ >, 1 I arc , _o, ,, 
(1 + -prf(B(m)) + W^- cxp(-B 2 (m)) 



q = l 



where 



d^ 



arc + (rn^) 2 
(2^)2 



l)erf(B(m)) 



1 I arc ,m£ „ , „ 2 , .. 
+ -a/ — J-l)exp(-B 2 (m)) 



x = / dCerf(B(m)). 

461 , 



B(m) = 



26) + m£ 
\/2arc 



(59) 
(60) 

(61) 



and this integrations can be performed directly. 
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Figure Captions 

FIG. 1. Phase diagram (a, 9) for the Q = 3 state network with uniformly distributed patterns and connectivity c as shown, at 

T — 0. The full and the long-dashed lines represent the retrieval and the thermodynamic transition, respectively. The latter 

ends on the right at the (dotted) optimal performance line and the spin-glass phase appears at the left of the paramagnetic 

phase boundary indicated by the short-dashed lines. The two retrieval regions I and II are discussed in the text and the inset 

corresponds to c = 0.63 where the distinction betweeen these phases starts to disappear. 

FIG. 2. Stable phase diagram to the right of the de Almeida- Thouless (dot-dashed) line in the (a, T) plane for the Q — 3 

state network with uniformly distributed patterns, 6 — 0.2 and connectivity c as shown. The full lines represent the retrieval 

transition. 

FIG. 3. Phase diagram (a, 6) for the Q = 4 state network with uniformly distributed patterns and connectivity (a)c = 0.5, 

(b)c = 0.48, (c)c = 0.1 and (d)c = 0.01, at T — 0. The full lines represent retrieval transitions and the optimal performance 

is indicated in dotted lines. The central region is the best performance phase FM\ and there is a low-a coexistence region 

between phases FMza and FMi. 

FIG. 4. Phase diagram (a, 9) for the Q = oo state network with uniformly distributed patterns and connectivity c as shown, at 

T — 0. The full lines represent retrieval transitions, the optimal performance is indicated in dotted lines and the onset of the 

binary network is shown by long-dashed lines. The phases are stable to the right of the de Almeida- Thouless line (dot-dashed). 

FIG. 5: Connectivity dependence for the maximum storage capacity a max and the corresponding 9 max in the Q = oo state 

network with uniformly distributed patterns at T — 0. 
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